Digital picture display system with photo clustering of camera roll and social media photos

ABSTRACT

A digital picture frame including a camera integrated with the frame, and a network connection module allowing the frame for direct contact and upload of photos from electronic devices or from a user&#39;s social media account or her or his community members&#39; social media accounts. The integrated camera is used to automatically determine an identity of a frame viewer, and a viewer profile automatically determined from the identity of the viewer. The displayed photos are automatically shown and/or changed according to the detected viewers.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/814,330, filed on 10 Mar. 2020, which is a continuation of U.S.patent application Ser. No. 16/156,354, filed on 10 Oct. 2018, which isa continuation of U.S. patent application Ser. No. 15/150,623, filed on10 May 2016, which is a continuation-in-part of U.S. patent applicationSer. No. 14/949,353, filed on 23 Nov. 2015, which is acontinuation-in-part of U.S. patent application Ser. No. 14/455,297,filed on 8 Aug. 2014, which is a continuation-in-part of each of: U.S.patent application Ser. No. 14/051,071, filed on 10 Oct. 2013, and U.S.patent application Ser. No. 14/051,089, filed on 10 Oct. 2013. Theco-pending parent application(s) are hereby incorporated by referenceherein in their entirety and are made a part hereof, including but notlimited to those portions which specifically appear hereinafter.

FIELD OF THE INVENTION

This invention relates generally to a digital picture frame and, moreparticularly, to a digital picture frame that is a Mobile PositionalSocial Media (MPSM) output device that automatically providespersonalized photo viewing in automated combination with a proximate orremote electronic device and/or an MPSM account.

BACKGROUND OF THE INVENTION

Existing digital picture frames synchronize with popular social mediaplatforms, such as Facebook™ and Instagram™, using, among othertechniques, a WiFi™ interconnection. However, multiple steps may berequired for such synchronization. These manual steps can be difficultto do if one wants to change to a different social media account or todirectly retrieve from a picture capture or storage device, such aswhile entertaining guests. There is a continuing need for improvementsfor implementing displays such as home digital picture frames.

SUMMARY OF THE INVENTION

The present invention provides a digital picture frame thatautomatically detects viewers and/or viewers' electronic devices, andautomatically displays photos or videos relevant to the detectedviewers. The relevant photos or videos are obtained upon detection fromthe electronic device and/or social media accounts of the detectedviewers. The photos or videos can also be automatically obtained fromsocial media accounts of one or more community members of the detectedviewer(s), such as photos taken and posted by a friend or relative ofthe detected viewer. The device and method of this invention providesfor improved and updated photo display based upon current photos and/orsocial media updates, and thus relevant to the viewer. The device ofthis invention can be used as an MPSM device to display any MPSMinformation, as desired, including but not limited to MPSM informationobtained according to methods described herein.

A general object of the invention can be attained, at least in part,through a digital picture frame including a digital display mountedwithin a frame, a camera connected to the frame, a network connectionmodule adapted to connect to an electronic device and/or social mediaaccount of a viewer viewing the digital picture frame and to receivephotos stored within the electronic device and/or the social mediaaccount, and an automated display module adapted to automatically changephotos displayed on the digital display to photos from the electronicdevice and/or social media account of the viewer, upon automaticdetection with the camera of the viewer.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes: automatically determining with the camera anidentity of a viewer of the digital picture frame; automaticallydetermining a viewer profile from the identity of the viewer; andautomatically changing photos shown on the digital display as a functionof the viewer profile upon detection of the viewer with the camera. Themethod can include automatically augmenting a photo display on thedigital display as a function of photo metadata and the viewer profile.In embodiments of this invention, the viewer profile is automaticallylearned and uploaded via an electronic device and/or a social mediaaccount of the viewer.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes automatically determining with the camera anidentity of a viewer of the digital picture frame, automaticallyconnecting to a picture capture or storage electronic device of theviewer over a network, such as a WiFi network, and automaticallydisplaying photos shown on the digital display obtained from theelectronic device over the network connection.

The device and method of this invention can automatically display aslideshow of photos relevant to one or more display viewers as afunction of photo context selected from time taken, photo location,and/or photo content. The device can automatically access, using thenetwork module, at least one electronic device and/or social mediaaccount of: the viewer and/or at least one social media community memberof the viewer. Photos from the one or more electronic devices or socialmedia accounts are desirably automatically loaded and displayed. Inembodiments of this invention, obtained photos are automaticallysequenced on the digital display as a function of profiling traitsselected from chronological order, photo location, photo activity,and/or community member.

The device and method can detect and recognize two or more different andsimultaneous, or otherwise present, persons viewing the digital pictureframe, and automatically change the photos displayed on the digitaldisplay to photos uploaded from an electronic device and/or a socialmedia account of each of two or more persons viewing the digital pictureframe. The photos from the electronic device or social media account ofeach of the two or more persons can include shared activity photos fromactivities shared by the two or more persons. The invention includesembodiments where a server computer automatically determines sharedactivity photos from the electronic device and/or social media accountof the each of the two or more persons as a function of contextinformation automatically associated with the shared activity photos bythe same or different server computer. The context information of eachof the shared activity photos can include, but is not limited to, aphoto location, a photo activity, and/or a present community member.

The invention still further comprehends a display device with one ormore of: automatic detection for configuration, namely theself-straightening or self-positioning on a vertical (e.g., wall) and/ora horizontal (e.g., table) platform; the automatic detection of viewingconditions, namely, based on the distance of the viewer, enlarging orcontracting the picture; based on the lighting, increasing or decreasingthe lighting intensity. The invention also comprehends a display devicethat: via face recognition, automatically detects the viewer and targetsthe photos for her or him based on their potentially learned profile;via the automatic identification and tagging of the location, activity,and community member involvement on a per photo or video, indicating thewith whom, where, when and what was being done when the picture wastaken as well as any other associated metadata; and/or providing a“story telling” capability. By storytelling, chronological stories,optionally simultaneously displayed on a split screen, are grouped by:purely time, namely in sequential or reverse chronological ordering;location, namely a traversal of sites on a location based trip;activity, namely in chronological ordering of a given or similar set ofactivities; community member involvement, namely a pictorial interactionwith community members, potentially segmented by particular communitymember or members; or any other profiling traits of a recognized userthat can be used to cluster or segment photos or videos for automaticstory telling.

The digital picture frame of embodiments of this invention operates inconcert with a method, system, and apparatus, such as embodied in anMPSM or other software application, that automatically determines andshares a location and/or an activity of a user. The application learnsuser activity over time, with the learning based upon user locationsand/or context. The present invention generally provides methods andapplications for an MPSM that automatically understands and informs the“who, what, when, where, and/or how” of a user and the user's community.For example, who are the user and their community with, what are theuser and their community doing, where geographically are the user andtheir community, when are, and historically when were, the user andtheir community doing this, and/or how can users' behaviors be modified?The method includes automatic tagging of photos taken by the users withcorresponding context data (e.g., location, activity, people present)learned by the system for use in determining photos to display on thedigital picture frame of this invention as discussed above.

The invention includes a method of learning and sharing locations and/oractivities, and pictures generated therefrom, of a user participating ina social networking service. The method is executed by a softwareapplication stored and executed on one or more computers or dataprocessor systems, such as a mobile device (e.g., phone, tablet, orlaptop) and/or an application server (such as for connecting usercommunities). In one embodiment the method includes receiving userinformation about a destination, automatically associating the userinformation with the destination, and automatically sharing the userinformation in the social networking service upon further user arrivalsat the destination prior to receiving any additional user information.

The invention still further includes a computer server for providing atracking and/or social networking service, such as operating the methodsdiscussed herein. The computer server of one embodiment of thisinvention includes a tagging module configured to correlate userinformation and/or pictures thereof to a user destination or activity, adatabase module configured to store user information including userlocations and user activities at the user locations, and a communicationmodule configured to automatically share a user activity in the socialnetworking service upon further user arrivals at a corresponding one ofthe user locations. The computer system can also include an associationmodule configured to associate the user activity with the correspondingone of the user locations.

In embodiments of this invention, the system or application identifieslocations, and over time, automatically “checks-in” not only thelocations but what the locations imply in terms of potential activitiesof the user. That is, given a location and a user, the system desirablysuggests what activity or activities the user typically partakes at thatlocation. For example, if a user frequents a location in “Potomac,” thislocation might be identified as “parents' home.” Furthermore, at thishome, a variety of activities might be common such as: “visitingparents,” “drinking tea,” “eating lunch,” or “sampling wine.” Inembodiments of this invention, each time the user appears at thatlocation, based on context, defined by elements of or surrounding theactivity such as but not limited to time of day, day of week,immediately preceding activities, weather, surrounding people, etc., aset of likely occurring activities are identified. The user can beprompted with a list from which to select a subset of these activitiesor to identify a new activity. The invention can also include ranking auser's potential suggested activity based on context and presenting thatranked list to the user, or the user's community. The inventiongenerally provides a learning component that can allow the manual inputsto become automatic prompts, which can become automatically issuednotifications for the location based upon the context. The prompts canbe issued through any known format, such as an application alert on thedevice or a text message to the user. The invention also supports theuser changing activities for a given location at any time, and/or userimplemented delay of the notification of a user's location or activity.

The invention can include the incorporation or creation of usercommunities and sub-communities, with such communities andsub-communities sharing information. Embodiments of the inventioninclude automatically identifying a user's location and activity, anddesirably notifying that user's community of that user's location, aswell as sharing relevant photos from the user's community to the digitalpicture frame. Particular embodiments of this invention provide one ormore additional community functionalities including, without limitation,automatically identifying a user's activity and notifying that user'scommunity of that user's activity, commenting on user activity andlocation report by user or community—with multiple and multimediacomments supported, supporting the “liking” of user activity by thecommunity, supporting the user tagging of location, activity, or thepairing of location and identity—tagging can be textual or via anymultimedia means, correlating the individual user's activity with theongoing activity of others within the community, and/or correlating theindividual user's activity with the past activities of others within thecommunity, all useful in generating user profiles and tagging photos forautomated display on the digital picture frame of this invention.

One embodiment of this invention provides a method of and system forautomated determining of locations and/or activities of a userparticipating in a social networking service. The method is executed byan MPSM computer system and automatically determines a positionaldestination of a user, automatically deduces as user information alocation type and/or user activity of the positional destination, andautomatically shares the user information as instructed in the socialnetworking service. Deducing the user information is based upon contextinformation about the positional destination, desirably with minimal orno input by the user. The context can include, without limitation,time-dependent information, past and/or current associated userinformation, past user and/or community information about the location,and/or third party information. The context can be used to at leastreduce location types and/or user activities, for example, as a functionof the past location type and/or user activity of the positionaldestination for a given time period.

Other objects and advantages will be apparent to those skilled in theart from the following detailed description taken in conjunction withthe appended claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIG. 2 shows a representative area of a user according to an MPSM methodof one embodiment of this invention.

FIG. 3 illustrates geofences surrounding a current reading and itsimmediate neighbors according to one embodiment of this invention.

FIG. 4 illustrates the determination of a location via intersectingcircles according to one embodiment of this invention.

FIG. 5 illustrates the processing flow employed to identify an arrivalaccording to one embodiment of this invention.

FIG. 6 illustrates a system view location summary of an individual useraccording to one embodiment of this invention.

FIG. 7 illustrates a system view transit summary of an individual useraccording to one embodiment of this invention.

FIG. 8 illustrates a system view activity summary of an individual useraccording to one embodiment of this invention.

FIG. 9 illustrates a system view time location breakdown in comparisonto other users according to one embodiment of this invention.

FIG. 10 illustrates a system view a listing of activities shared withother users according to one embodiment of this invention.

FIG. 11 illustrates a view presented to users that quantifies theirshared patterns according to one embodiment of this invention.

FIG. 12 illustrates a view presented to users summarizing their weeklybehavior according to one embodiment of this invention.

FIG. 13 illustrates a view presented to a user of her/his and her/hiscommunity's activities in a pictogram format, according to oneembodiment of this invention.

FIGS. 14-16 illustrate various pictogram summaries for user activities.

FIGS. 17-19 illustrate adding photos to pictograms and changingpictograms.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a digital picture frame that interactswith either a picture capture or storage device, referred to herein asan electronic device, running a digital picture frame application and/orsocial media to automatically display digital photographs from theelectronic device and/or a social media account of one or more personsviewing the picture frame. As used herein, the terms “picture”,“photograph”, or “photo” refer to or include digital still photographsor digital video, including a combination of few-second animationseither or both before and after a still frame, unless otherwise noted.The photos can be obtained by social media users and stored in theusers' social media accounts. When a social media user views the digitalpicture frame, the digital picture frame automatically recognizes thatuser and automatically loads photos relevant to that viewer from her orhis electronic device and/or social media account and/or from at leastone electronic device and/or social media account of her or his socialmedia community member(s). For example, relevant photos from a communitymember's social media account may include the viewer or be from a sharedactivity that the viewer attended with the community member. Automatedsocial media activity learning methods, as described herein, are used toautomatically determine relevancy of photos to one or more communitymembers. Community members may choose to make photos shared or privateto avoid personal photos being shared.

FIG. 1 illustrates a digital picture frame 20 according to oneembodiment of this invention. The digital picture frame 20 includes adigital display 22 mounted within an edge frame 24. The digital display22 can incorporate any suitable screen, such as, without limitation, anLED screen or a touch screen, as are known and commercially available. Acamera 25 is connected to and integrated in the edge frame 24. Thepicture frame 20 includes a network connection module that connects overa wired and/or wireless network to a social media server computer 26 andcan automatically load photos stored within one or more social mediaaccounts accessed through the server 26. The picture frame 20 furtherincludes an automated display module in combination with the networkconnection module and adapted to automatically change photos displayedon the digital display 22 to photos from an electronic device and/or asocial media account of the viewer and/or one or more community members,such as upon automatic detection with the camera of the viewer. Thedigital picture frame 20 desirably includes a microphone 27 incombination with the automated display module and/or network connectionmodule, such as for receiving spoken instructions from a viewer tochange the displayed photos.

The digital picture frame 20 further includes a leveling device 28connected to or otherwise integrated with the frame 24. The levelingdevice 28 automatically detects when the frame is not level. Properpositioning of the digital picture frame is important to achieving usersatisfaction. Auto detection of the positioning in comparison to theangle of the displayed photograph illustrated can be achieved using anyof the image alignment or realignment techniques known in the art. Inembodiments of this invention, the leveling device 28 can transmit ordisplay corrective measures for physical correction.

In another embodiment of the invention, the digital picture frame 20 isconfigured with a leveling device that includes one or more motors 23,such as paired attached with a corresponding wall bracket via gearing.The motors 23 can be incorporated in a wall hanging mechanism as shownin FIG. 1, or otherwise implemented with respect to a frame stand. Upondetection of an unleveled viewing as determined by either the levelingdevice 28 or any other mechanism, the dual motors 23 activate andautomatically mechanically operate as needed to level the digitalpicture frame 20, such as a vertical adjustment on wall mountinghardware, via the adjustable hanging mechanism or bracket.

The digital picture frame 20 includes an automated display adjustmentdevice in detection combination with the camera 25. The automateddisplay adjustment device automatically detects frame environment and/orviewer position using the camera 25 and automatically adjusts settingsof the digital display 22. Augmented with the camera 25, the digitalpicture frame 20 supports a host of optional, additional capabilities.Using conventional, known in the art, image processing techniques,enhancement of the picture quality can be automatically made. Forexample, due to an automatically detected viewing angle or environmentalbrightness, picture lighting intensity and contrast can be altered. Thatis, depending on conditions, the brightness and contrast can beautomatically modified to provide better viewing. Additionally,depending on the distance and angle of the viewer, the size of the imagecan be enlarged, possibly focusing on the center or content rich area ofthe picture. Similarly, a close viewer may wish to have the photographor video in full scope, or maybe, even side-by-side with one or moreother, possibly related pictures or videos.

In one embodiment, as shown in FIG. 1, the digital picture frame 20includes the capability to ‘daisy chain’ one or more additional frames20′. That is, all participating frames can be connected so as to portraya single story, either chronologically or thematically, allow for twoviewers, and/or provide a composite view of a single image. The frame 20can include a network connection 10, such as an audio-video outletand/or network outlet, to connect to a corresponding inlet 12 of thesecond frame 20′. The network connection between frames can be by wiredconnection (e.g., HDMI cord or Ethernet cable) or wireless communication(e.g., Bluetooth or WiFi) The frame 20 can optionally include a poweroutput, such as outlet 14 to receive a power cord 16 of the second frame20′, so as to require only a single power source.

An embodiment of a digital picture frame 20 further includes multiplepower options. In addition to conventional outlet and battery options,the frame can include at least one photoelectric cell. Inductive, orwireless, charging can also be incorporated.

The digital picture frame 20 includes a facial recognition module withany suitable facial recognition software, in combination with the camera25 and automated display module. Face recognition software can bedeployed to automatically detect the identity of the viewer(s). It isalso within the scope of this invention to identify the viewer via thelocation or proximity of their personal electronic devices such as butnot limited to their mobile phones or tablets. By identifying theviewer(s) the automated display module can then specially target, namelyselect, prioritize, and/or order the photos or videos displayed for theidentified viewer.

Identifying the viewer enables a variety of novel applications. Inembodiments of this invention, a viewer analysis module is combined witha facial recognition module to determine, for example, one of mood orhealth of the viewer upon detection of the viewer with the facialrecognition module. The automated display module can augment a photodisplay on the digital display as a function of the determined mood orheath of the viewer, such as by showing photos to increase the viewer'smood.

Longitudinal analysis of the viewer is possible. Such analysis supports,for example, health commentary. By analyzing the viewer's face, commentssuch as, but not limited to, “you seem to have gained weight”, “you looktired”, or “you seem pale; do you feel well?” can be announced to theviewer. Similarly, “mirror, mirror on the wall” remarks such as “youlook good today” can be stated, potentially elevating the viewer's mood.Displaying identified mood enhancing photographs, such as, for example,ones that include far away loved ones can likewise be shown to furtherelevate the viewer's mood. Identification of mood enhancing photographscan be automatically accomplished, for example, via the recognition of a“smiling face” or spoken words of the viewer when the picture waspreviously shown, and tagging of the photo.

Viewer identification also enables photo selection personalization. Aviewer personalization module can be operatively combined with thefacial recognition module, whereby the viewer personalization moduleautomatically identifies preferred photos or restricted photos as afunction of viewer information upon detection of the viewer with thefacial recognition module. Photos favored by the viewer (e.g.,grandchildren for the grandparent) can be selected, and converselyphotos containing, for example, ex-spouses or in-laws can be repressed.Likewise, identifying viewer characteristics, such as age, enables thelimiting of photos displayed. For example, children might not be shownphotos containing potentially inappropriate content, such as containingnudity, or particular activities or individuals, such as relatives thathave recently passed away. Thus, it is within the scope of thisinvention to limit or specifically select photographs to be displayedbased on relationships of the viewer with individuals in thephotographs, such as ex-spouses; characteristics of individuals shown inthe photographs, such as babies; characteristics of the viewer, such asage; activities illustrated in the photographs, such as smoking;locations displayed in photographs both physically described (e.g.,Chicago) or relatively described (e.g., viewer's place of residence); orbased on any characteristics that can be determined via image analysisor photograph metadata included. The filtration or selection ofphotographs based on criteria is determined by settings that can be set.

Additional specialty settings are included. Such setting place the framein nightlight mode where only a dim glow is illustrated; photo recordmode to simply record all in the line of vision; video chat mode withother devices; and “stranger in the house” mode where only limited orstock photos are shown to prevent the potential identification ofresidents.

The digital picture frame 20 can likewise be used as a viewer assistant.Multiple ailments or conditions, such as but not limited to Alzheimer's,strokes, and dementia, affect memory. Thus, the digital picture frame 20can be used to remind the viewer of the identity of family members andfriends as well as of occasions, locations, activities, etc., ofrelevance as the photos displayed are labeled both by content and/or bymetadata.

As previously discussed, the network connection module connects over anetwork connection to the social media server computer 26. It canlikewise connect either directly or indirectly to the mobile device 29′of any community member or interested party having an appropriatelyconfigured device, such as digital picture frame application specificsoftware. The connection need not be within a geographic proximity, andit is likewise within the scope of this invention that a geographicallyremote connection is supported via networks such as but not limited toas the Internet.

The social media server computer 26 can be any one or more computersthat implement any suitable social media platform, such as the MPSMplatform and automated learning steps described herein. The servercomputer 26 can obtain or have access to photos and learn user oractivity information associated with each photo from, for example,mobile devices 29 of a plurality of MPSM users or community members,such as according to methods described herein. In embodiments of thisinvention, the automated display module augments a photo display on thedigital display 22 as a function of photo metadata from the social mediaaccount stored on the server computer 26. The photos are automaticallydownloaded from the server 26 to the digital frame 20 upon, for example,automated detection of a corresponding viewer(s).

In embodiments of this invention, the photos can be changed according tothe learned bias of a viewer, expressed and stored as part of a userprofile at the server computer 26 and/or the digital picture frame 20.The picture frame 20, such as via the automated display module, includesor obtains the viewer profile of the viewer upon detection of the viewervia the camera. By establishing a viewing profile for each known orexpected viewer, e.g., if a viewer is or has a community member,individual or group user viewing preferences can be maintained. Aprofile can be established using any one or more of the many knownprofile techniques known in the art, including but not limited to,asking the viewer, learning based on previous user selections, and/orcorrelations with previously automatically learned, via MPSM methodsherein, locations, activities, and/or community member interactions.

In a preferred embodiment, viewer profiles are automatically learnedaccording to methods described herein, and uploaded via an electronicdevice and/or the social media account of a detected viewer and/or heror his community members. The automated display module desirablydisplays a slideshow of photos for a detected viewer that is uploadedfrom: an electronic device of the viewer, and/or a social media accountof the viewer or one or more social media community member of theviewer.

In the digital picture frame embodiments of this invention pictures orvideos displayed can be augmented with the automatically learned andidentified metadata corresponding to the pictures or videos. That is,the information learned, captured and/or displayed identifies thelocation, activity, and community member involvement on a per photobasis, indicating the with whom, where, when and what was being donewhen the picture or video was taken as well as any other associatedmetadata, such as the camera or video equipment used to capture thephoto or video. Slideshow of photos for a viewer can be assembled andshown in a display order automatically determined as a function of photoprofiling traits such as time (e.g., chronological order), photolocation, photo content (e.g., people or activity shown in the photo),and/or community member presence at the time/location that a photo wastaken.

The invention includes a method of displaying photos on a digitalpicture frame, such as shown in FIG. 1. The digital picture frame ishung or set on a surface by a user, for explanation purpose, namedSusan. The picture frame automatically determines with the camera whenSusan is viewing, or is the primary viewer of, the picture frame. Thepicture frame automatically determines Susan's profile from herdetection. In embodiments of this invention, Susan's viewer profile isloaded from Susan's electronic device and/or social media account, andmay be supplemented by additional information gathered via the pictureframe itself, such as face recognition, account access information,and/or manual entry (e.g., picture frame display preferences). Theviewer profile is desirably created and continually updatedautomatically according to the automated learning methods describedherein. The viewer profile can include information such as, withoutlimitation, family information, community member information, and/oruser favorites (e.g., favorite locations, activities, colors, flowers,animals, etc.), learned through the social media account.

When the camera and coordinated software determines Susan is viewing thepicture frame, e.g., either directly or merely in the vicinity, thepicture frame automatically displays photos shown on the digital displayas a function of the viewer profile information. The photos displayedare automatically augmented or picked as a function of comparing photometadata and the viewer profile. In embodiments of this invention, thepicture frame automatically displays for Susan a slideshow of photos(still and/or videos) relevant to Susan as a function of photo context.The photo context can be selected from the time the photo was taken(e.g., all photos from July 2015), photo location (e.g., all photostaken in Chicago), and/or photo content (e.g., all photos of vacationsor involving her sister and/or her friend Mary). The photos can beimported primarily from her electronic device and/or social media, andoptionally supplemented from photos on a recordable medium of the frame(e.g., hard drive or inserted flash drive or memory card) or photosobtained from a third party web site.

It is within the scope of this invention to prompt the user, possiblyvia any communication mechanism including but not limited to text andvoice commands, to determine the exact wishes of the user. If thedigital picture frame is additionally equipped with a microphone 27 andcorresponding speech processing software, voice commands can likewise beprocessed using any of the many known speech processing systems. Susancan select a context, such as by spoken instructions, to be display. Asan example, Susan may request all photos related to “college”, relatedto “vacations”, including “family”, or including or relating to aparticular person. The digital picture frame will download relevantphotos for display from one or more electronic devices and/or socialmedia accounts, of Susan's and/or a community member of Susan, that havecontext matching the request. If Susan asks for pictures of her and hersister, the pictures may include her sister or have been taken in thepresence of both her and her sister, such as a picture of the EiffelTower taken during a trip with her sister to France. The Eiffel Towerphoto is identified by context metadata that identifies that both Susanand her sister were present when the photo was taken. The contextinformation is desirably automatically learned and stored with thepicture in the social media account of Susan, using the learning methodsdescribed herein. Alternatively, the Eiffel Tower photo can be part ofthe sister's social media account, and because her sister is a communitymember and has given sharing rights, the social media server computerautomatically shares the photo to Susan's digital picture frame.Desirably, any photo relevant to Susan can be automatically shared fromany of Susan's community members and automatically shown on the pictureframe display.

In one embodiment of this invention, the digital picture frame accessessocial media or her electronic device without detailed instructions fromSusan and loads and sequences photos displayed on the digital display asa function of profiling traits selected from chronological order, photolocation, photo activity, and/or community member. Having automaticallylearned the location, activity, and community member involvement on aper photo or video basis using the method described herein, a “storytelling” capability is supported. That is, in storytelling,chronological stories, optionally simultaneously displayed on a splitscreen, are grouped by: purely time, namely in sequential chronologicalordering; location, namely a traversal of sites on a location basedtrip; activity, namely in chronological ordering of a given or similarset of activities; community member involvement, namely a pictorialinteraction with community members, potentially segmented by particularcommunity member or members; or any other profiling trait(s) of arecognized user that can be used to cluster or segment photos forautomatic story telling.

In one embodiment of this invention, Susan's friend Mary joins Susan inviewing the picture frame, either directly or within a detectablevicinity of the picture frame. The picture frame automatically detectsthe additional presence of Mary, automatically detecting her identity,automatically obtains or loads Mary's profile, and automatically changesthe photos displayed on the digital display to those relevant to bothSusan and Mary. The photos could be obtained from Susan's social mediaaccount(s) or personal picture capture or storage electronic device butpreferably, the photos are obtained from both Susan's and Mary's socialmedia accounts and/or local personal picture capture or storageelectronic devices, e.g., mobile phones 29 and 29′. Desirably the photosfrom each of the two viewers are shared activity photos; from activitiesshared by the two friends. The frame 20 and/or server computer 26automatically determines shared activity photos of the two persons as afunction of the learned context information automatically associatedwith the shared activity photos by the server computer. As more peopleare automatically detected, photos from more accounts or devices can beautomatically added, and the photos automatically organized by contextsuch as photo location, a photo activity, and the present communitymembers. Options can be added on the social media accounts to share ornot share photos with community members or their devices according tothis invention.

In another embodiment, Mary and Susan each simultaneously, or withinsome specified duration of time, look at their respective digitalpicture frame 20 and 20′, such as linked as shown in FIG. 1 or networkconnected when each frame in a different location (e.g., theirrespective residences). Remote frames can communicate, via a network,such as through server 26, and indicate to both Mary and Susan, each attheir respective frames 20 and 20′, their respective presence,potentially indicating when Mary and Susan are both viewing or viewed aphotograph from a preselected set of photographs within the specifiedduration of time. The duration of time can be defined in terms of aspecific time or as within a time interval commencing from when eitherMary or Susan viewed the given photograph from within the preselectedset. The frames 20 and 20′ can automatically or upon instructioncoordinate so that the two are viewing the same photos. Likewise thephotos can be augmented to included shared experiences, and be uploadedfrom both of Mary and Susan's linked devices 29 and 29′ and/orrespective social media account.

A non-limiting embodiment of this invention consists of at least onepicture capture or storage electronic device 29 communicating directlyvia a network or wired or other wireless connection with the digitalpicture frame 20. To support such interaction, the picture capture orstorage device 29 connects via the network to the digital picture frame20 and downloads the digital picture frame interaction software to thedevice 29. Once installed, the application software scans all theavailable pictures on the device. The remaining process description isillustrated using a non-limiting example based on face selection. It islikewise, however, within the scope of this invention to select based onobjects other than faces including but not limited to scenes, materials,and activities. A second device 29′ (or more) can likewise link to theframe upon being in connection proximity.

Once selected, all photos containing faces are clustered by theapplication. Individual photographs can then be automatically filteredbased on features. Some of these features are quality related, e.g.,red-eyes or blurred; some features are content related, e.g., excludingcertain activities, people, or locations. Those photographs that remainselected are tagged with additional metadata. Metadata includes allphotograph generated information, such as but not limited to location,time, or weather, as well as derived data, such as but not limited tocluster identity.

Another embodiment of this invention includes, and the digital pictureframe is implemented with, a method, system, and/or apparatus, such asembodied in an MPSM or other software application, that automaticallydetermines and shares a location, an activity, and/or photos of a user.The application learns user activity over time, with the learning basedupon user locations and/or context. The application can learn throughautomatically determining activities at locations based upon knowncontext information and past context information for the location. Theapplication can tag photos for determining context relevancy for showingon the digital picture frame, as discussed above. The invention furtherincludes energy saving location methods for the mobile device that canbe used to more efficiently allow the location and social media aspectsof the invention to be implemented on a mobile device. The method andapplication can be used for any suitable function, such as a safetyand/or reminder serves, and is particularly useful for use in socialmedia applications and for generating photos for display on the digitalpicture frame. The invention will be described below with implementationin an MPSM system, and particularly with an MPSM application that learnsuser activity over time, with the learning based upon user locationsand/or context.

The MPSM method and system of this invention is mobile and positional innature. Such systems, like many other systems originally developed onone type of computing platform but migrated to another, operate not onlyon mobile environments. That is, while MPSM implementations are targetedto primarily execute on mobile devices, such as but not limited tosmart-phones, tablets, and/or laptops, they often support implementationfor non-mobile environments such as but not limited to desktops andworkstations, servers, and large scale compute farms and cloud computingservers. The invention will be described below with a mobile device,such as smart phone having cell service, a GPS system, and access to theInternet via WiFi.

The MPSM method and system of this invention is desirably executed orimplemented on and/or through a mobile device computing platform. Suchcomputing platforms generally include a processor, a recordable medium,an input/output (I/O) device, and a network interface capable ofconnecting either directly or indirectly to the Internet. The mobiledevice executes over a networked environment, a non-limiting exampleshown in FIG. 2. The mobile device is connected, either directly orindirectly, using any of the many techniques and technologies known inthe art, over a network, to back-end system or systems,itself/themselves computing devices. The mobile device can connect witha remote server, shown in FIG. 2 as server 38, to store and/or accessuser or community information.

MPSM systems are used to support users remaining socially aware of theircommunity. That is, their primary usage typically is to actively monitorthe location and activity of family members, friends, colleagues, andgenerally others within one's community. Communities can be partitionedinto sub-communities where the union of the sub-communities forms theuser's community. The sub-communities may or may not overlap. Thepartitioning of communities into sub-communities is beneficial insupporting specialized applications. For example, while a user mighthave general interest in the location and activity of all of theircommunity members, they might be particularly interested in the locationand activity of those who might be suddenly in need of assistance.

The creation of a community can include the issuing of invitations. Aninvitation is a request by a user A of another user B to allow theinviting user, user A, to track the activities of the invited user, userB, and vice versa. If the invited user accepts, the inviting and invitedusers form a community.

A community is relevant to only that user which formed it. That is,different users have different communities. A community is a grouping ofinvited (referred to as remote) users by the inviting (referred to aslocal) user. A local user can partition or merge a community, thusforming a sub-community or a parent community, respectively. Forexample, consider 5 users: Bob, Sam, Sally, Alice, and Susan. Bob caninvite Sam, Sally, and Alice, thus forming his user community. Bob canlikewise partition his community into a sub-community consisting of onlySam and Sally. Sally can invite Susan. Thus, Sally's community wouldinclude Bob (via his invitation) as well as Susan. If no additionalinvites occurred, Sam's and Alice's respective communities would onlyinclude Bob (each via Bob's invitation), while Susan's community wouldonly include Sally (via Sally's invitation).

Providing users with the opportunity to expand their communities in aconvenient manner is advantageous. Such expansion can seamlessly beaccommodated by including users listed in a user's contact lists eitheras a whole or selectively into their community. Contact lists include,but are not limited to, users listed in a user's local address book,e-mail contact list, Twitter follow list, LinkedIn connections list,and/or Facebook friends list. By incorporating users listed in a user'scontact list, the user's community is expanded without effort. Note,however, that selected inclusion can be supported; thus enablingcommunity growth without unnecessarily over-expanding the community.That is, entries from the contact list can be included in their entiretyand the user can selectively remove those entries which s/he wishes tobe excluded from the community. Similarly, entries from the contact listcan be selectively added.

Users are identified by their account identifier. To use MPSM a useraccount is created. User accounts generally require a user login, whichis a unique user identifier, and a password or equivalent. After havingcreated an account, a user can log in. Initially, the local user doesnot have a community. In embodiments of this invention, over time, themethod and application tracks the activities and location of the localuser. Should the local user establish a community as aforementioneddescribed, the community members will likewise be tracked. Local usersreceive notifications of the location and activities of their communitymembers. Once logged in, the local user can select to activate ordeactivate self and community tracking and notification. If notoverwritten, default settings are used.

Whenever logged in and tracking is enabled, a user's location andactivity is tracked. That is, a user periodically records their locationand/or activity. Locations are tagged by name Names can be but are notlimited to the following schemes: physical (e.g., 123 Oak St.), absolute(e.g., Acme Coffee), and/or relative (e.g., my work office), orproximity (e.g., two miles from home). Activities are typically events.These events might be common to the entire community such as: “drinkingcoffee,” “eating lunch,” “sampling wine,” “working from home,”“commuting,” etc., to more specific to a local user such as “restoringcar” or “driving to lake home.” Multiple activities can occursimultaneously. Users can change their activities at any time.

Unless preloaded or derived from an external source, such as but notlimited to a location database, initially, all locations and activitiesare unknown. Local users must record all such location-activitycombinations, i.e., a local user must name or tag the location and theassociated activity. A list of activities common to the local user'scommunity can be provided. This community activity list can be rankedeither arbitrarily (randomly), according to most recently used, mostfrequently used, relevance to location, alphabetically, etc. Eventually,an activity list specific to the local user is learned. This local useractivity list can be displayed to the local user either individually,along with the community list, or merged with the community list. Again,any of these lists can be ranked as previously mentioned.

FIG. 2 illustrates a representative area 30 to demonstrate a method ofand application for locations and/or activities of a user participatingin a social networking service. The area 30 is shown as a cellularcommunication network including a plurality of cells 32 each disposedaround a cellular communication antennae or base station 36. Within thearea are a plurality of destinations each shown as including a WiFiInternet connection. The local user has one or more electronic devices,such as a mobile device that communicates with a remote server 38 viathe cellular network and/or the WiFi connections. As will be appreciatedthe methods and applications of this invention can operate within anysuitable size and configuration of the communication area, depending onwhat the user encounters.

Destination 40 is the home of the user. The user commutes to office 42for work on most business days. On the way the user typically stops atthe coffee shop 41. For lunch on most days, the user visits restaurant43, but on Wednesdays the user typically meets a second user for lunchat restaurant 44.

At each destination 40-44, the user enters user information about thedestination. The application and computer system that receives the userinformation automatically associates the user information with thedestination, and stores the user information in a locations database,such as on the device and/or at server 38. The destination desirably isdetermined automatically and tagged with the user information, such as alocation name of the destination and/or the user activity beingperformed at the destination. For example, destination 40 can be taggedas “home” and likely has numerous activities associated with it. Thedestination 41, and any photos taken, can be tagged as its establishmentname “Acme Coffee” or simply “coffee shop” and associated with the useractivity of “buying coffee” or “latte time.” The manually entered userinformation can then be automatically shared to the user's community ina social networking service. Similar user information is received forthe other destinations 42-44. The user information desirably includesany other information about the location or activity, whether manuallyentered or automatically determined, such as the time of the visit oractivity. Some destinations, such as home or work will likely havemultiple user activities over a period of time, such as “coffee break,”“meeting time,” and/or “quitting time.”

The computer system receives user information and associates the userinformation with the corresponding destination, and any photos taken,for multiple visits to each of the destinations 40-44. The computersystem begins learning the locations and user activities. In embodimentsof this invention, the user can be automatically prompted forconfirmation of the user information upon arriving at a destination toconfirm the location and/or user activity. For example, the user can beprovided with an automatically generated list of previously entered useractivities for the destination upon arrival, thereby promoting efficientcollection of information. The items on the list can be listed in anorder based upon a particular ranking, such as number of times enteredpreviously or based upon a context, such as what activity is likelybeing performed at a particular time of a particular day.

Over time, the computer system learns the user information and beginsautomatically associating and identifying at least some user activitiesfor corresponding locations and any photos taken. As will beappreciated, the automatic identifying of activities at locations willlikely occur at different rates for different activities and locations,with some locations having fewer activities and/or more frequent visitsthan others. In preferred embodiments of this invention, the systemautomatically shares the user information in a social networking serviceupon automatically detecting further user arrivals at the destination.Photos taken are likewise automatically tagged with the userinformation. The automatic sharing of user locations and/or activitiesdesirably occurs upon the user's arrival at the location, or at aparticular time at the location. As such the invention includes anautomatic detection of the user's arrival at a destination. Theautomatic sharing and photo tagging desirably operates without useraction and prior to receiving any additional user information for thedestination.

As an example, the user may typically purchase lunch at destination 43,but on Wednesdays typically goes to lunch with a friend or spouse atdestination 44. The lunch routines of the user, and particularly theWednesday lunch routine, can be learned by the system and automaticallyshared to the user's community upon the system automatically determiningarrival, without manually input from the user. If the user is havinglunch with a community member, then the system can automaticallydetermine that both users are at the same location together toautomatically recognize and confirm the lunch activity, and proceed toautomatically share the information for both user's to their respectivecommunities. If the user deviates from a routine, the system can knowthis, and refrain from sharing the typical destination, by the mobiledevice detecting a different location than the typical routinedestination.

In embodiments of this invention, learning is accomplished by any knownmachine learning, data mining, and/or statistical techniques known inthe art. Supervised, semi-supervised, and/or unsupervised approaches canbe used, including, but not limited to Naïve Bayes, Neural Networks,Support Vector Machine, and/or Associating Mining based techniques.

The MPSM method and system of this invention desirably records allposted locations and activities. Throughout use, the disclosed inventionlearns the corresponding locations and the set of associated activities.More so, via comments made by the local user and by the local user'scommunities, the importance of the activities can be learned, such asfor the prompting discussed above. Importance can be either local useror community biased. Additionally, importance can be biased by context.For example, community members as a whole might prefer “eating steak,”“eating pizza,” and “eating sushi,” in that respective order. On theother hand, a local user might only eat sushi. Thus, local user biaswill yield “eating sushi” only, while community bias will suggest“eating steak,” “eating pizza,” and “eating sushi,” in that respectiveorder.

In embodiments of the MPSM method and system of this invention,locations are named according to a naming convention. Regardless of thenaming convention used, a location is a physical geographical position.More so, physical geographic locations associate properties that canvary with or be dependent on context, namely time and date (hours, dayof week, calendar date, etc.), users involved, and their relationshipsto each other, etc. This context can affect the associated location nameor activity.

A common scheme that can be used to at least assist in identifying aphysical geographical location is via the use of geocoding. Geocoding isthe representation of a physical location via the pairing of latitudinaland longitudinal coordinates commonly referred to as a lat-long pair.Global Positioning Systems (GPS) can also determine a physical positioncoordinated via the triangulation of satellite transmissions. TypicallyGPS devices derive lat-long pairs which are made available to a varietyof applications, often via map displays. GPS economics, accuracy, andsimplicity of use resulted in their wide appeal and commercial success.Their continuous use in mobile devices is problematic, however, as theyare energy intensive and rapidly drain the battery. Thus, alternativemeans or approaches to detect locations are desired.

Embodiments of the MPSM method and system of this invention, asdiscussed above in FIG. 2, use or rely upon cell coordinates. Whenmobile devices communicate with a cell tower, they send their cellcoordinates. These coordinates are recorded by the cell provider and aretypically not publicly known. The cell phone or, in this case, themobile device supporting the positional social media system, however, isaware of their coordinates. Thus, the device can store the cellcoordinate position and automatically associate that cell coordinatewith the location name provided by the local user. Over time, a locationdatabase of cell coordinate and named location pairs is created. Thelocal portion of the database favors the local user. The union of allthe local portions of the location database desirably constitutes thename space of the entire MPSM system of this invention. It is understoodthat any of the many database management systems or storage schemesknown in the art can serve as the platform for this location database.Thus, location names can be provided without the need to rely on aglobal positioning system, reducing battery consumption. Location datacan additionally or alternatively be purchased or otherwise provided bya third party.

An additional and/or alternative approach for automatic locationdetermination relies on WiFi triangulations. Mobile devices can grow andmaintain a database of known open WiFi networks, for clarity we callthis database an Open-WiFi-Net database. Such mobile devices can use theinformation stored or derived from the information stored in theOpen-WiFi-Net database to further refine the accuracy of a locationwithout the use of GPS. Via point triangulation, when an Open-WiFi-Netdatabase is available, the mobile operating system uses not only thecell tower but also WiFi triangulations to determine location. It iswithin the scope of this invention to use either or both cell-phone andWiFi triangulations to enhance location information in addition to anyother disclosed approach. The mobile device can use the WiFi signal at adestination, such as destination 43, and additionally or alternativelyany detectable open WiFi signal from a neighboring location, such asestablishment 45 that is adjacent destination 43.

Having created the location database, searching, namely querying, thedatabase uses the cell coordinate or the location name That is, alocation name query takes a location name as input and returns thecorresponding cell coordinate. A cell coordinate query takes a locationname as input and returns the corresponding location name Note that,multiple names can be attributed to a given cell coordinate. That is, alocal user might name a location using multiple different names;different users can name same locations using different names.Similarly, the same name can be used for different cell coordinatelocations. All names corresponding to a given cell coordinate arereturned. It is within the scope of this invention to selectively returnnames based on context, user, or community bias. Similarly, all cellcoordinates corresponding to a given name are returned. Again, it iswithin the scope of this invention to selectively return coordinatesbased on context, user, or community bias. Ranking of the resultsreturned can, when desired, be biased towards the local user.

A key concern for MPSM systems is collecting location information.Clearly any location information available within the mobile deviceshould be harnessed. Thus, if GPS readings or any other locationinformation is generated by other device resident applications, thesereadings are desirably recorded and utilized by the method andapplication of this invention. However, reliance on strictly otherapplications to obtain positional information is obviously not realisticor possible.

In embodiments of the MPSM method and system of this invention,positional information is obtained via the use of geofences. A geofenceis geographical boundary or “fence” surrounding a positional reading. Asthese boundaries are radius based, geofences are generally circular.Location transmission occurs whenever a handover of one cell tower toanother occurs and is expected but not guaranteed to occur once ageofence boundary is crossed. To track location, periodic locationtransmissions are required. Since location transmissions must beminimized to conserve device energy, transmissions should only occurgiven geographical movement. Thus, crossing a geofence should generatesuch a transmission. Unfortunately, as crossing a geofence does notguarantee a location transmission, increasing the likelihood of atransmission is necessary.

In contrast to the known uses that surround a location with a singlegeofence, to increase the likelihood of a location transmission duringmovement, embodiments of this invention include surrounding a locationgeofence with a plurality of geofences. In one embodiment of thisinvention, a method and system of tracking a user includes determining alocation of the mobile user, automatically establishing a first geofencearound the location, and automatically establishing a plurality ofadditional geofences around the first geofence, with each geofenceincluding a boundary. A location transmission is obtained by the mobiledevice upon crossing a boundary of the first geofence or any of theplurality of additional geofences. Multiple neighboring geofences areadvantageous since they increase the likelihood of a locationtransmission as their boundaries are likewise likely to be crossed givenmovement.

FIG. 3 representatively illustrates a geofence 60 surrounding a currentlocation 62. The geofence 60 is surrounded by additional geofences 64,all within a given cellular tower transmissions cell 65. Note that partof a neighboring geofence 64′ is not fully within the cell 65, andhence, limits its benefits since a cell tower handoff by movement intocell 65′ will generate a location transmission.

Geofences are implemented as software processes. Operating systems formobile devices, such as but not limited to iOS and Android, limit thenumber of processes available to an application, and thus, the number ofgeofences is bounded. However, this limit typically exceeds the numberof geofences generated using the approach described above. Therefore,additional processes are available, and hence, additional geofences arepossible.

To increase the likelihood of a location transmission given movement, inembodiments of the invention, the remaining available processesimplement static geofences. A static geofence is not dynamicallygenerated given a new location as previously described. Rather, a staticgeofence is one that is fixed and represents those locations that arelikely to be crossed by a given user. That is, users are habitual andfrequent a limited set of locations often, for example but not limitedto, their home, office, or favorite wine or sushi bar. By learning thefrequent locations of users both individually and system wide andsetting static geofences at these locations, biasing by the individualuser, the probability of a location transmission is increased sinceadditional geofences are likely crossed.

More so, these repeated locations vary by city, county, state, country,etc., as well as by other factors such as but not limited to day andtime. Geographical and temporal presence can thus be used to vary theset of static geofences for a given user. For example, the set of staticgeofences for a given user will vary if the user is in Washington, D.C.rather than in San Francisco, Calif. Similarly, the set of staticgeofences for a given user will vary depending on the day and time. Forexample, a user frequents work on weekday mornings but frequents theirfavorite bagel shop on Sunday mornings and their favorite sushi bar onThursday evenings.

Location transmissions suffer from a margin of error. Thus, it isdifficult to precisely pinpoint and tag a location with a singletransmission. Embodiments of this invention include automatic refiningof a location of a user destination as a function of user routines, suchas established by several user visits to the destination. As timeprogresses however, and a user frequents the same location multipletimes, multiple location transmissions for the same location arerecorded. In one embodiment of this invention, as representatively shownin FIG. 4, by overlapping the transmitted location along with its marginof error, a more accurate location can be derived. The overlapping oflocation transmissions for a given location 70 between streets 72 andwithin geofence 74, along with their margin of errors, represented ascircles 76, yields an accurate location placement.

As shown in FIG. 4, location accuracy improves as related data arecollected. Related data, however, can, at times, be somewhat erroneous(in terms of accuracy). A non-limiting example is an entrance to ashopping mall. Such an entrance is not necessarily at the center of thecomplex. Regardless of the entrance displacement from the center of thecomplex, the entrance location can still be used to increase locationaccuracy of the mall complex since the readings for the entrance areconsistent. That is, for a given user, given mobile device, givencarrier, etc., such location recordings remain consistent, all be it,slightly erroneous. Thus, even dirty, namely potentially inaccurate,data can result in correct location identification.

Additionally, having established a location, corresponding lat-long paircoordinates can be reversed engineered, namely mapped back onto, a placename. These derived lat-long pair coordinates become yet an additionalinformation component that is used by a learning system to better refinea mapping to a named place. Machine learning, data mining, andstatistical approaches that are supervised, semi-supervised, orunsupervised can be used, as known in the art, to cross-correlate allavailable location related data.

Once determined, the user information including the location and/or theuser activities are automatically stored in a database. Embodiments ofthe MPSM method and system of this invention include a computer serverfor providing and implementing the tracking and/or social networkingservice of this invention. The computer server includes a locationmodule to determine the user location and/or a tagging module configuredto correlate manually entered user information to a user destination anda database module configured to store user information including userlocations and user activities at the user locations. For social mediaand photo sharing, the server further desirably includes a communicationmodule configured to automatically share a user activity or photo in thesocial networking service upon further user arrivals at a correspondingone of the user or community locations. The server can also include anassociation module configured to associate the user activity with thecorresponding user location and any photo taken.

Since location transmissions are needed during movement, the obviousquestion arises: when should the transmissions cease? That is, thesystem must determine when the user has arrived at a location to knowwhen to perform the automatic steps discussed above. As discussed above,GPS systems are an energy drain on a mobile device, particularly as theGPS remains on and linked with the satellites to maintain locationdetection. Keeping a GPS application operating is a drain on both theprocessor and the battery of the mobile device. This invention providesa method and executable application that conserves energy by notcontinually running during use of the mobile device.

Embodiments of the MPSM method of this invention provide an automatedmethod of tracking a mobile user that includes providing a locationmodule configured to receive location transmissions, placing thelocation module into a sleep mode, awakening the location module uponreceipt of a location transmission, and determining a location with thelocation module. These placing, awakening, and determining steps arerepeated, thereby placing the application into a sleep mode when notneeded, thereby reducing the drain on the mobile device. The applicationgoes into sleep mode when necessary or when desired, such as when theapplication is not needed, e.g., during extended movement or upon anarrival at a location. In embodiments of the MPSM method and system ofthis invention, the application can go into sleep mode whenever a timesince the device awakening exceeds a predetermined time allocation, orupon a determined rate of travel exceeding a predetermined threshold,thereby indicating extended travel.

FIG. 5 illustrates one exemplary, and non-limiting, method according toan embodiment of this invention to automatically detect arrival at adestination. The method is useful for tracking a user's location for anyof various reasons, including, for example, for safety, to provideautomated reminders, and/or to provide automated suggestions to the userbased upon the destination and/or surrounding area. The method of FIG. 5is particularly useful for implementing the method and system discussedabove, and can be used to implement other applications and method toprovide energy savings compared to GPS location methods in mobiledevices.

FIG. 5 includes a flow chart 100 that includes and/or represents threedistinct situations, namely, an actual arrival, rapid movement, andsporadic movement without an actual arrival. Initially, the applicationis in sleep mode. Sleep mode is a state when no processing, and hence noenergy consumption, takes place. Processing occurs once the applicationis awoken. A location transmission, such as a cell tower transmission oranother application obtaining location information, awakens theapplication in step 102. Since the application awakening occurs due to alocation transmission, the current location is known.

Once awakened, the application typically has a maximum amount of time tocomplete its processing. This limit, called time allotment, is set bythe device operating system. All processing must complete prior toexceeding the time allotment. Ideally, the application should relinquishthe processing flag back to the device operating system before theoperating system forcefully removes the application from its activequeue. Voluntarily terminating an application, namely returning it tothe sleep mode, rather than having it forcefully terminated by the hostoperating system, is consider good citizenship. In step 104, theapplication initializes two timers, namely, a timer count representingthe duration of time the process has executed since last awakening, anda stationary count representing the duration of time since the lastdetected device movement.

As time progresses and the process executes, the timer count isincremented in step 106. In one embodiment of this invention, wheneverthe application processing time exceeds the operating system timeallocation (108 —YES branch), the application is voluntarily placed insleep mode 105. Note that the time allocation threshold is notnecessary, but set to support good citizenship.

Assuming that the time limit has not been reached (108—NO branch), theapplication waits for t time units in step 110. After waiting t timeunits, new current location data are obtained is step 112 and storedlocally on the device in step 114. In step 116, the current location iscompared to the previously known location. If the two locations differ(116—NO branch), the rate of travel is computed in 118. If the rate oftravel exceeds a threshold (120—YES branch), the process is desirablyand voluntarily placed in sleep mode 122. Rapid travel is unlikely toresult in an immediate or near term arrival; thus, checking locationswhile moving rapidly unnecessarily uses device energy. Eventually, theapplication process is awoken with the device moving at a slower rate.At that time, location checking is needed as an arrival might soonoccur. If or when the rate of travel is slow (120—NO branch), movementis noted in step 124, and the loop is repeated commencing with theindication that additional processing time has elapsed in step 106.

Thus far, the arrival detection process has been voluntarily placed insleep mode either due to having exceeded the self-imposed processingallotment quota which is desirably set just slightly below the operatingsystem's time limit that leads to the removal of the application fromthe active queue (108—YES branch) or having travelled too rapidly(120—YES branch). Slow travel has resulted in simply recording thelocations traveled, noting the movement exists in step 124, and awaitingeither arrival or process termination.

Arrival is determined when the same location is detected for asufficient duration of time. That is, an arrival is potentiallydetermined when the location remains the same (116—YES branch). Thestationary detection count is then incremented in step 126. If thestationary threshold is not yet exceeded (128—NO branch), theapplication waits for t time units in step 110, and the current locationis obtained in step 112 and stored locally in step 114. A sufficient andpredetermined duration at the same location eventually surpass thearrival detection threshold (128—YES Branch).

Once arrival is determined, arrival is declared in step 130, all dataregarding the prior locations visited and stored locally are compressedand sent to the back end system supporting the application in step 132.A new location checkpoint is established in step 134, and the process isplaced in sleep mode 136. From the sleep modes, the process of FIG. 5repeats upon a known location.

Compression of location data is typically performed prior to localdevice to back-end system transmission as often the location data manynot be needed at the back end. Location data may not be needed in cases,for example but not limited, during rapid travel. Although exemplifiedas having data compression occur prior to the sending of the data to theback-end, it is within the scope of this invention to compress locationdata prior to storing them locally.

All parameters described above for FIG. 5, for example t (for the timeunits), timer count, etc., are system and device dependent.Experimentation with and fine tuning of these and other parameters iswithin the scope of this invention. Also within the scope of thisinvention is the tuning of these and other parameters via the use ofmachine learning, data mining, and statistical approaches; supervised,semi-supervised, and unsupervised approaches can be used.

As discussed above, once the user has arrived at a destination, thelocation identification, user activities at the location, and/or anyproximate third party members of the user's community are determined, ifnot already known. In this way, the devices automatically continuallydetermine locations which can be used to identify any establishmentsand/or any community members at or within proximity to the location.

User activities are actions or events. Example user activities includebut are not limited to “drinking wine,” “flying,” “reading,” “attendingconference,” or “commuting.” Users specify a particular user activityeither by selecting from a provided list or by entering a different useractivity. As discussed above, the provided list is generated by storingall previously entered user activities from all systems users butbiasing the ranking of the provided activities based on context, thelocal user, their community, or a combination thereof.

All location and user activity pairs are stored in a databasecorrelating the location with the activity. Any of the many databasemanagement systems or storage schemes known in the art can serve as theplatform for this location-activity database. Furthermore, it is wellunderstood in the art that the location-activity database can store manyadditional features. For example, the user identity and date and time ofthe pair are recorded.

Over time, the database grows and contains a sufficient number of pairsto support mining. The volume of data needed to mine correlations isdependent on the mining algorithm deployed and the level of accuracyneeded. As known in the art, there are many machine learning, datamining, and statistical approaches to support mining By using any of themany available such approaches, either individually or in combination, alocal user activity preference per location is learned. Example learningapproaches include supervised, semi-supervised, and unsupervisedapproaches including but not limited to Naïve Bayes, Neural Networks,Support Vector Machine, and Associating Mining based techniques. The useof proprietary mining techniques is likewise within the scope of thisinvention. Once local user preference is learned, this preference isused to bias the aforementioned provided user activity list.

There are many approaches to identify locations. Automated locationidentification is accomplished by periodic checking of the currentlocation. Periodicity of checking depends on, for example, the methodused to determine the location, the desired frequency of reporting,recording, and notification, and the resources available to support thechecking. Other periodicity determination approaches known in the artcan likewise be used. One approach to automate location identificationis the periodic determination of lat-long pairs via the use of a GPSdevice. An online service or a locally resident database can be used tocorrelate the GPS readings with locations. A preferred embodiment ofthis invention uses the aforementioned location database. Whenever atransmission to a connected cell tower is made, the cell coordinates ofthe transmitting device are used as a search query against the locationdatabase. If a match is detected, that location is identified. Anotherpreferred embodiment detects locations upon the crossing of geofenceboundaries as previously discussed. Note that both dynamicallydetermined geofence boundaries and static geofence boundaries detect alocation. Yet another preferred embodiment detects locations bycapitalizing on location transmissions generated by any otherapplication operating on the mobile device requesting locationinformation.

In embodiments of the MPSM method and system of this invention, localusers, unless disabled by a local user, can be provided with automatednotifications for themselves and for their community members. Thesenotifications describe locations, activities, or correlated locationsand activities for themselves and their community members. For example,unless disabled by the user, any time a user arrives at a new location,the local user and their communities can be notified of the user's newlocation. Automated location detection and notification, unlessdisabled, occurs without requiring a local user prompt.

Similarly, activity notification can be automated. Once a user arrivesat a location, a set of activities previously occurring at that locationis shared with the community or provided to the local user forinformation or sharing. If the user chooses to confirm at least one ofthese past activities, both the local user and their respectivecommunity members are notified of this at least one activity, and anyphoto taken is automatically tagged with the context information.

In another embodiment of this invention, automated notification involvesshared experiences. A shared experience is one that associates multipleusers. These associations can be passive or active. A passiveassociation is strictly informative in nature while an activeassociation requests an action. Non-limiting examples of passive sharedexperiences based on locations include: “User A is at User A's office,as is User B” and “User A is at home as is User C.” Note that the firstexample involves multiple users at the same physical location, namelyUser A's office, while the second example involves multiple users at thesame relative locations, namely their homes, but at different physicallocations.

Similarly, passive shared experience notifications can be based on useractivity. Non-limiting examples of passive shared experiences based onactivity include: “User A is eating lunch as is User B” and “User A isparticipating in her favorite sport as is User B.” Note that the firstexample involves multiple users participating in the same activity,namely eating lunch, while the second example involves multiple usersinvolved in similar nature of activities, namely participating in theirown favorite sport, which can be different actual activities, namelyracquetball and swimming In both passive shared experiences based onlocation and on activity, known in the art machine learning, datamining, and statistical approaches that are supervised, semi-supervised,or unsupervised approaches can be used to correlate relative locationsand activities to physical locations and activities.

Other shared experiences can prompt for action, and are thus consideredactive. A non-limiting example of an active shared experience promptingfor action includes: “User A posted a picture when at Penn Station; youare now at Penn Station; please post a better picture?” Thus, activeshared experiences request the user to actively react. As above, activeshared experiences can be location or activity based and can be absoluteor relative. Note that it is likewise within the scope of this inventionthat individual user notifications be active and passive, in a similarmanner as described above. However, the correlation of locations andactivities both for passive and active are based strictly on thecurrent, past, or projected expected activities of the individual userrather than those of multiple users.

Typically, only changed locations and activities are notified. That is,a location or activity is not typically repeatedly identified. However,a local user can request repetitive notifications based on anytriggering condition known in the art.

Local users do not always remember to indicate a new location name orconfirm which of the possible suggested name or names the systemindicated for the given the location. As such, it is at timesadvantageous to prompt the local user for information. However, overlyaggressive prompting might annoy the user. In embodiments of thisinvention, the application non-invasively prompts the user upondetecting an unknown location for the given local user. To avoidannoyance, prompting is repeated only rarely, say twice; the number ofrepeated prompts can be set as a parameter. Similarly, to provide asense of comfort, if the back-end system recognizes the location basedon the local user's community members' naming schemes, it prompts thelocal user with guiding messages, for example but not limited to “Manyof your community members call this location The Tasting Room”.

Identification of activities associated with a given location or a givencommunity member can be additionally or alternatively automaticallyinferred in multiple ways. In embodiments of this invention, thecomputer system can automatically determine a positional destination ofa user, such as by using a mobile device discussed herein, andautomatically deduce as user information a location type and/or useractivity of the positional destination. The user information can bededuced, at least in part, based upon the destination context. Exemplarycontext information includes, without limitation, time-dependentinformation (e.g., what time of day is it?), community information(e.g., who is also there?), and/or third-party information about thepositional destination. This method, tied with automatic sharing of theuser information in a social networking service, can provide a partiallyor fully automated process for determining user location and activity,and tagging photos taken with the context information.

In one embodiment of the MPSM method and system of this invention, theautomatic deducing of the user information is based upon known orlearned user routine. As discussed above, local users typically followstandard routines. Some routines are daily, weekly, monthly, etc. Otherroutines are context dependent. Regardless of the nature of the routine,learning via any of the many statistical, machine learning, data mining,or business analytical techniques known in the art, enables predictivebehavior and automated activity and location suggestion. For example,but not limited to, if a local user always goes out to lunch at noon onevery weekday, then if an unknown location is detected on a Tuesday atnoon, then the application can suggest that this unknown location islikely a restaurant and the activity is likely eating lunch. Similarly,routine identification enables the prevention of transmissions bothpositional and informational. For example, but not limited to, if alocal user always goes to sleep at midnight on Sunday through Thursdayand awakens at 7:00 am the following day, then energy can be saved ifthe application voluntarily places itself in sleep mode during the hoursthat the local user is known to be sleeping. Additionally, routines caninvolve a sequence of activities and locations. A non-limiting exampleof a sequence of activities includes: On weekdays, Eric arrives at hisoffice at 8:00 am, drinks coffee at 10:00, develops software from 11:00am until 5:00 pm, commutes home at 5:30, and finally, arrives at home at6:00 μm.

Another location and/or activity deduction approach is by association.The automated deducing can include automatically associating a user witha second user at a positional destination. If the second user's locationand/or second user's activity is known, then the system canautomatically infer the location type and/or user activity of the firstuser from the second user location and/or activity. Consider a previousknown event such as: “Community member Sally swimming at the Somersetpool”, assuming that the Somerset pool location was previouslyidentified. As an example of automatically determining a currentactivity of community user Sam, the system identifies through locationdetermination that Sam is currently at the same location as Sally, andalso that Sally is currently at the Somerset pool. From thisinformation, possible automatically postulated associations andactivities are: “Sam is at the Somerset pool”, “Sally is swimming”, and“Sam is swimming”. Thus, it is possible to infer an activity for acommunity member from association with another community member. It iswithin the scope of this invention to use any logical inference methodsknown in the art to generate plausible associations. It is also withinthe scope of this invention to obtain confirmation of the plausiblepostulated activity by the community member, in this case Sam, by askingeither Sam or Sally or by any other means known in the art.

Desirably the computer system operating the MPSM automatically storespast user information, including past location type and/or user activityof the positional destinations of all users. User information for futurevisits to repeat positional destinations can be automatically deduced asa function of the stored past location type and/or user activity of thepositional destination. In embodiments of this invention, the system canrely on recorded previous activities of a user, a community member, orany system user at a given location to postulate on a user's activity ata given location. Past context information for past visits to thepositional destination by the user and/or community members of the usercan be compared to a current context of the user's visit to thepositional destination to deduce the user information. In oneembodiment, the system can reduce possible location types and/or useractivities as a function of the past location type and/or user activityof the positional destination.

As an example, at a given Location A, users previously studied, talked,ate, and drank. Thus, if a user's positional destination is detected asat Location A, then plausible activities postulated can be studying,talking, eating, and drinking. More so, if the given user's communitymembers only previously talked, ate, and drank, it is with a higherprobability to postulate that the given user is talking, eating, anddrinking rather than studying. Furthermore, if the given user visitedLocation A previously, but only talked and drank, then an even higherprobability is that the user is currently talking and drinking ratherthan eating and studying. It is within the scope of this invention topostulate some or all of the previously detected activities of a givenlocation. More so, it is within the scope of this invention to rankorder the activity suggestions according to the relevance of thepreviously visiting users to the given current user. As previouslydescribed, the system can request confirmation of suggested activitiesthrough the user's mobile device.

The system can additionally or alternatively reduce possible locationtypes and/or user activities as a function of the past location typeand/or user activity of the positional destination as a function of thetime of day. The system can rank possible location types and/or useractivities of the positional destination based upon known past timeperiods corresponding to the time of day of the current user visit. Forexample, again given Location A, if previous visiting users wererecorded to study one or more times during the intervals: 3:00-4:30 PMand 7:30-9:00 PM, and to drink one or more times during the intervals:4:00-7:00 PM and 8:30 PM-2:00 AM, then a current visiting user atLocation A at 3:15 PM is likely studying, at 4:15 PM is likely to beeither studying or drinking, and at 1:00 AM is likely to be drinking.More so, if the given user's community members only studied between3:15-4:30 PM then it is with a higher probability to postulate that thegiven user is studying rather than drinking at 4:15 PM. Furthermore, ifthe given user visited Location A previously but only studied, then aneven higher probability is that the user when at Location A is studying.It is within the scope of this invention to postulate some or all of thepreviously detected activities of a given location. More so, it iswithin the scope of this invention to rank order the activitysuggestions according to the relevance of the previously visiting usersto the given current user. As previously described, the system canrequest confirmation of suggested activities through the user's mobiledevice.

In embodiments of this invention, time context alone can be used topostulate activities. For example, if most days, a user is recorded tobe drinking coffee between 9:00-10:00 AM, then, without contradictoryinformation, a plausible activity postulate is that at 9:35, the user'sactivity is drinking coffee. Again, as previously disclosed, it iswithin the scope of this invention to rank order the postulated activitysuggestions according to the relevance of the previous users to thegiven current user and/or to obtain confirmation of suggestedactivities.

Additionally, it is also within the scope of this invention to rankorder the time postulates based on frequency of occurrence within thetime interval. This rank ordering applies to both location based andlocation independent time based postulates. For example, if in theinterval 4:00-4:30 PM, community: members studied 25 times but drank 5times then, at 4:15, it is with a higher probability to postulate thatthe given user is studying rather than drinking.

In embodiments of the MPSM method and system of this invention, thesystem can search and/or use, if available, external, third partyinformation about the positional destination for postulating activitiesfor a given location. For example, third party vendors might provide,either free of charge or for a fee, activity information for a givensite. Consider a marketing website of a centralized homepage for agrocery store chain. Such websites are likely to contain addresses ofmany or all of the associated stores. Since these stores all supportshopping, an activity associated with these locations is shopping.Similar information can be derived or purchased from other sources suchas but not limited to commercial information repositories. Additionally,maps can be parsed. Given a location of a road, an activity of thatlocation is likely to be driving. Various and alternative third partyinformation gathering approaches and their incorporation into activityclassification and postulation can be incorporated into the method andsystem of this invention.

Suggested activity information, particularly but not limited toinformation obtained or derived from third party vendors, might beadditive or might be contradictory. Thus, combining or reconcilingpotential activities is needed. The use of voting schemes, biased basedon credibility of the source or on frequency, such as majority, or otherknown techniques, can be incorporated in the method and system of thisinvention. Note that differing suggested plausible activities mayadditive or may be contradictory. The use of techniques such as, but notlimited to, conflict resolution methods, ontology determination, andclustering, etc., can be incorporated to recognize potential conflictsand to expand classification is within the scope of this invention.

Additionally, the classification of plausible activities based onactivities occurring in the surrounding vicinity is likewise within thescope of this invention. For example, consider an unknown locationadjacent to two known locations, such as, but not limited to, twoneighboring stores or two neighboring beaches. For the neighboringstores, known activities might include shopping and strolling, while forthe neighboring beaches, known activities might include sunbathing andswimming Given location proximity, it is within the scope of thisinvention to suggest a user's activity at the unknown location to beeither shopping and strolling or sunbathing and swimming, respectively.Confirmation can always be obtained for suggested activities and to biassuggested activities based on user familiarity and frequency ofoccurrence.

In embodiments of the MPSM method and system of this invention, localusers can opt to delay their notifications. That is, once a location isvisited or an activity occurs, a local user can opt to have thenotification of the location or activity be delayed by some period oftime. Delaying a notification provides the local user with the abilityto notify their community of the location visit or activity occurrence,but still provides the local user time to progress to the next locationor activity. As discussed above, users can also choose to automaticallyshare or not share photos taken with the digital picture frames of thisinvention.

Notifications can be complemented with correlations with other communitymembers. That is, both the local user and their respective community canbe automatically notified with a comparison. A comparison, for examplebut not limited to, can identify other community members havingpreviously conducted a specific activity or having visited a givenlocation previously. Comparisons are made by checking other communitymember locations and activities against those of the local user.Checking is performed via a search of the location-activity database. Ifa match exists within a specified or predetermined period of time, acomparison notification is made automatically. The period of time can bearbitrarily set or can follow some logical time quantum such as hour,day, week, etc.

Locations and activities are known by name. However, in addition to aname, locations and activities can have associated personal labels.Labeling locations and activities can detail familiarity to the locationand activity. User labels for locations can be surrogate names, forexample, “favorite city” for Chicago, can be songs or sound waves, forexample song words “my kind of town, Chicago is” for Chicago, can be apicture, for example “the Water Tower” for Chicago, can be a video, forexample “a panoramic view of the Chicago skyline” for Chicago, or anycombination of these and other multimedia tags supported by the localdevice. Similarly, user labels can exist for activities. For example,“favorite vice” for drinking wine, or it can be a song or sound wave,for example the song words “a bottle of red” for drinking wine, or itcan be a picture, for example, a wine bottle picture for drinking wine,or it can be a video, for example “a panoramic view of a vineyard” fordrinking wine, or any combination of these and other multimedia tagginglabels supported by the local device.

In embodiments of the MPSM method and system of this invention, localusers and community members can comment on their own and each other'slocations and activities. Comments can take any of the many multimediaforms provided by the local device. These include, but are not limitedto, text, sound, pictures, videos, or any combination thereof. Multiplecomments can be made by the local user, their community, or combinationthereof. In addition to stating their opinions (commenting), communitymembers can prompt for clarification. That is, by issuing “what”comments, community members request additional information on the postedlocations and activities. Additionally, user can “like” their own andeach other's locations and activities. By “liking” a location oractivity, community members express their satisfaction of theirrespective community members' presence in terms of location andactivity. Multiple community members as well as the local user can“like” a location and activity.

The MPSM method and systems of this invention can track vast data onboth the local user and their respective community members. These datacover, including but not limited to, locations, activities, and alsoindividuals both who are system users and those who are not. These datacan be stored and summarized. A summary of the local user and communitymember locations, activities, time durations involved in each of theselocations and activities, individuals who they encountered, etc., can becomputed and presented to the user. This summarization can range fromsimple statistical aggregation to advanced correlations as derived byany of the many, both individually and combined, machine learning, datamining, business analytics, and statistical techniques known in the art.

Information that can be aggregated or derived can answer, exemplary butnot limiting, questions such as: how much time a local user spent doingthings, such as, working at home, working out, walking the dog,commuting to work?; how much time a particular community member spentdoing things, such as, working at home, working out, walking the dog,commuting to work? (Note that the information derived for the communitymember is based strictly on the information that that particularcommunity member chose to share); who are the five most commonindividuals that a particular user interacts with?; what is thelikelihood that after seeing a particular user, the given local userwould see a particular different individual?; which activities andlocations are most closely associated with each other and when are theymost likely to occur?; which three users among a given community aremost likely to visit a particular location together?

Local users can be provided with summaries of their locations, durationsat these locations, and activities at these locations. Furthermore, atthe discretion of the local user, these summaries are made available totheir community members.

The system can also generate and maintain both aggregation and derivedinformation. This information can be used to optimize suggestions toavoid obstacles, for example, but not limited to preferred routing ofcommuting path, promoted target advertising, for example but not limitedto location of nearby ice cream store for those users who frequentlyrecord “eating ice cream” as an activity, and a host of otherinformational services known in the art.

The following examples illustrate, without limitation, the abovediscussed data capturing, storing, analyzing, mining, and presentingMPSM functionalities of this invention. It is to be understood that allchanges that come within the spirit of the invention are desired to beprotected and thus the invention is not to be construed as limited bythese examples.

FIG. 6 illustrates a location summary of an individual user. As shown,two boxes are presented. The top box is a summary of where and how auser spent their last two weekend days, while the bottom box provides asummary of where and how a user spent their last five weekday days. Asshown, in both cases, the duration of time spent in a given location islisted. For example, looking at the top window, the user spent about 9hours in Georgetown in Washington and about 1 hour in O'Hare in Chicagoduring the last two weekend days. The user was obviously on travel asthe user spent less than a minute at home during the weekend.

FIG. 7 illustrates a user's transit summary, which is a summary of theuser's transit characterized by speed, namely slow, medium, and fasttravel, and when, where, and for how long did this travel occur, namely,duration and initial and terminating locations. The average speed islikewise noted. For example, the user traveled fast from Denver airportto home, a distance of 964.57 miles at an average speed of 291.29 MPH,and it took roughly 3 hours.

FIG. 8 illustrates a user's own activity summary That is, a summary ofthe user's weekly activity is provided that includes the frequency ofand percentage and absolute time involved in the activity over the pastweek. For example, the user was at the Four Seasons twice within theweek for 3.4% of their reported time or about 4 hours.

FIG. 9 illustrates a user's time summary in comparison to their friends.The summary of the user's time breakdown is made in comparison to otherswithin their community. For example, the percentage of time the userspent at home as compared to that of their friends is roughly 14%(0.14×) versus in transit which is 1.53 times as much.

FIG. 10 illustrates a summary of a user's activities shared with others.The summary of the user's activities is shown providing an indication ofthe amount of time and activities jointly experienced. For example, theuser jointly had sushi with and visited Ophir at his home. Alsoindicated in the bottom portion of the frame are the names of one'scommunity members that the user did not see (top listing) or interactedwith (bottom listing) in the past week.

FIGS. 11 and 12 illustrate views presented to the user of a quantifiedtogether board and of a quantified individual board, respectively. FIGS.11 and 12 represent information in a more pleasing form for presentingto the user. FIG. 11 summarizes time spent and activities experiencedtogether. For example, two users spent 73 hours together this weekeating, working, and running. The last time they had drinks together wastwo weeks ago (bottom right boxes). The last time they were together was3 days ago (top left box). FIG. 12 summarizes a user's week in review.The week in review highlights the main activities and locations (tophalf). Likewise summarized are other behavior patterns includinganomalies (bottom half).

Embodiments of this invention incorporate picture characters orpictograms into the mobile positional social media domain. The methodand system of this invention allow users to define, develop,incorporate, modify, classify and/or transmit pictograms, such asrepresenting user locations and/or activities, to community members andto global users. In one embodiment, a user of an MPSM device can selector create new one or more pictograms specific to themselves, to theircommunity, or globally, that is, to any user that has access to that oneor many particular pictograms. Users can define or redefine existingmeanings of each pictogram; a user can incorporate existing, namelyalready created and defined pictogram, into their messages; a user canassociate a picture to a given pictogram; a user can modify existingpictograms both for local and for her/his community; and/or a user canalso classify a pictogram as to its type, for example, but not limitedto, mood, activity, location, etc.

Any suitable pictogram can be added and/or used in the method and systemof this invention. One exemplary pictogram is known as Emoji. Emoji is acommonly used term that generally means picture characters or pictogram.Some Emoji representations are mapped onto Unicode representation andare thus available for use in a variety of desktop and mobile deviceapplications including the invention disclosed herein.

Using the location and/or activity determination methods discussedherein, the MPSM can determine, and possibly announce, activitiesoccurring, having occurred, or are scheduled or likely to occur usingpictograms. The pictograms can also be used to associate activities witha given location; the location of interest either being, will be, or waspreviously visited by a community member or is of relevance to a userrequest.

In one embodiment of this invention, a method and system of sharinglocations and/or activities of a user participating in an MPSM includesthe system receiving a user-defined pictogram for a destination andautomatically associating the pictogram with the destination and/or anyphotos taken. The method desirably also includes the automatic sharingof the pictogram with the photos and/or within the social networkingservice upon further user arrivals at the destination prior to receivingany additional user information. The pictogram desirably corresponds toa user activity at the destination, and can be manually selected by theuser from a list of predetermined pictograms or other photos ordrawings, etc.

Embodiments of this invention include a system that learns to associatethe pictogram with the destination and photos taken upon further visits.The pictogram can be automatically presented to the user through amobile device upon reaching the destination for confirmation and/orchanging to or selecting a new pictogram. These steps can occur forseveral visits to the destination, with the goal for automated learningand ultimately to provide an automated sharing of the pictogram for thedestination and/or with any photos. Where several pictograms have beenassociated with a destination, the several pictograms can be presentedover time for confirmation, preferably in a ranked list. In addition,the method and system can automatically determining one or more of theplurality of pictograms to share at a further arrival at thedestinations as a function of an automatically determined context of thefurther arrival, such as based upon a time of day or the presence offellow community members also at the destination, as discussed furtherherein.

The pictograms according to this invention allow for efficient pictogramsummaries of user and community activities and/or locations for anypredetermined time period, such as a day, week, month, and/or year. FIG.13 is a mobile device GUI that shows users' timelines of activities viapictograms. The pictogram timeline 140 includes pictograms that providevisualizations of a person's and their community's daily activities. Theuser activities are visually compared to what others are doing. The userand the community are represented by photographs and their names. Themost recent activity is represented by a pictogram 150 on the horizontaltimeline 140 that is closest to the user pictogram 142. The remainingpictograms 150′ for each user are showing in order of newest to oldest.The timeline can also display the timing 152 of the most recentpictogram. Feedback by others on the user's activity is also provided.In FIG. 13, one comment is currently available, as shown by the commentnotification 155.

In embodiments of this invention, photographs or other data can beassociated with the pictograms. In FIG. 13, a camera icon 154 can beassociated with, such as by overlapping, any pictogram to show when oneor more photographs or other data items are associated with thepictogram. Community users can touch or click the pictogram to displaythe associated photograph. The camera icon at the bottom of the screencan be used by the user to associate or change photos. The pictogramscan optionally be shown with the corresponding photos on the digitalpicture frame.

In another embodiment of the MPSM method of this invention, users can begrouped by common current or latest activity. As shown in FIG. 13, thefirst two community members display the same ‘house’ pictogram,indicating a common activity, albeit not necessarily at the samelocation (i.e., each is that their own home).

FIGS. 14-16 illustrate alternative pictogram summaries of a user'slocations and/or activities. FIG. 14 illustrates a summary of a user'sdaily activities via pictograms. The visualization summary shows theindividual's activities by pictogram for several days. FIG. 15illustrates a pictogram summary of a user's activities on amonth-by-month basis. FIG. 16 illustrates a histogram summary of auser's monthly activities aggregated by type and presented as avisualization of an individual's monthly activities.

FIGS. 17-19 illustrate an exemplary embodiment associating pictures withthe pictograms. FIG. 17 illustrates the option to take a photograph andassociate it with a selected pictogram. Additionally, it is possible toannotate the image with text using the annotating tool at the bottom ofthe GUI. To assist the user, the system shows or suggests a pictogramrepresentation. The selection of a new pictogram is possible if desiredor if the system suggestion is incorrect. FIG. 18 illustrates the optionto select a new pictogram to associate with the newly taken photo. FIG.18 occurs upon tapping the picture in FIG. 17. Two alternative sectionsexist in FIG. 18: a first section of suggested pictograms, such as basedupon learned location information, and a second section with pictogramswhich are globally available. The system postulates a pictogram toassociate with the picture using any of the many learning classificationalgorithms discussed herein and/or known in the art. FIG. 19 illustratesa system generated suggestion of the location associated with thephotograph. The user has the option to accept or modify the annotation.

This invention further provides a means to rapidly display and reviewuser photographs. In one embodiment, the device, such as through anexecuted application, includes and executes code instructions forproviding rapid feedback in an ergonomically convenient, intuitive userinterface. The code allows for displaying user photographs or other datain response to a plurality of user hand movements within a continuoususer swipe on a mobile device screen. User hand movements are eithercaptured via a touch screen or via gestures captured via the line ofview of an included camera.

By a user interface method, referred to as the Ownbey-scrub, or “scrub”for short, a user can hold down an icon to select and swipe through aclustered collection of pictures representing activities that occurwithin a given time span, location, with specific other users, or anyother clustering condition. Clustering of items can be accomplished inany of the many known in the art clustering techniques.

In embodiments of this invention, the scrub operates by placing a finger(and holding it) on an icon presented on a touch screen and scrubbing(without lifting off the screen) with that finger horizontally (left orright) or vertically (up or down). In response to the direction of thecontinuous finger contact/movement, quick feedback is provided to theuser in the form of rapidly displayed pictures. Left or right and up ordown represent forwards or backwards depending on user preference. Userpreference might differ due to any orientation (left or right handed) orany physical or logical conditions. By default, right to left and top tobottom is ascending order while left to right and bottom to top isdescending order. Composition movements consisting of both vertical andhorizontal can also be used. For example, a user might push and holdher/his thumb and scrub right to left, without lifting the hold,scrolling rapidly through her/his daily events from morning to night.Determination of direction of scrolling can be accomplished using anyknown in the art vector composition methods. Scaling the tracking of thescrub movement based on the number of pictures in the intended scrolland determining when such movement was performed can be done via anyknown in the art computations as used for such processing in the imageanalysis, graphics, and user interface domains. Additionally, it iswithin the scope of the invention to support voice activated commandsthat support similar scrub operations.

The invention further includes ranking the order the pictures arepresented within each scrub, and/or ranking the order the presentationof the pictograms associated with each user. Rank ordering can be basedon any of the many ordering criteria disclosed herein and/or known inthe art including but not limited to chronological time associated withthe set of activities, location of the activities ordered based on biasof locations, proximity to other users—either a specific user or anyusers within a user's community, proximity to a location where the lastphotograph was taken, and so forth.

Thus, the invention provides a digital picture frame including a cameraconnected to the frame, and a network connection module for use as adevice for displaying pictures from a user's electronic device and/orsocial media account or her or his community members' social mediaaccounts. The frame allows for efficient, automated access to photosrelevant to the viewer(s) of the frame. The automated frame allows forchanging photos for the viewer(s) without multiple manual steps.

The invention illustratively disclosed herein suitably may be practicedin the absence of any element, part, step, component, or ingredientwhich is not specifically disclosed herein.

While in the foregoing detailed description this invention has beendescribed in relation to certain preferred embodiments thereof, and manydetails have been set forth for purposes of illustration, it will beapparent to those skilled in the art that the invention is susceptibleto additional embodiments and that certain of the details describedherein can be varied considerably without departing from the basicprinciples of the invention.

What is claimed is:
 1. A digital picture display system, comprising: adigital photo frame including a digital display, a network connectionmodule adapted to connect to user electronic devices and to receivephotos stored within the user electronic devices, and an automateddisplay module adapted to automatically display photos and change photosdisplayed on the digital display from the user electronic devices; afirst interaction application stored on a first electronic device of afirst user, the first electronic device including a first digital photocollection, and the first interaction application including a firstplurality of automatically created digital photo clusters according tocontent from the first digital photo collection; a second interactionapplication stored on a second electronic device of a second user, thesecond electronic device including a second digital photo collection,and the second interaction application including a second plurality ofautomatically created digital photo clusters according to content fromthe second digital photo collection; wherein the first interactionapplication is configured to automatically obtain social media photosfrom a social media account linked to the first user and add the socialmedia photos to the first plurality of automatically created digitalphoto clusters according to matching photo content; wherein the secondinteraction application is configured to automatically obtain furthersocial media photos from a second social media account linked to thesecond user and add the further social media photos to the secondplurality of automatically created digital photo clusters according tomatching photo content; wherein the network connection module receivesdigital photos from one or more of the first and second plurality ofautomatically created digital photo clusters for display on the digitaldisplay.
 2. The digital picture display system of claim 1, wherein eachof the first interaction application and the second interactionapplication is configured to automatically obtain additional socialmedia photos from additional social media accounts of one or morecommunity members of the first user or the second user.
 3. The digitalpicture display system of claim 1, wherein the network connection modulefilters digital photos from the second plurality of automaticallycreated digital photo clusters to remove photos showing individuals notin community with eh first user.
 4. The digital picture display systemof claim 1, wherein the network connection module filters digital photosfrom the second plurality of automatically created digital photoclusters to only show shared activity photos from activities shared bythe first and second users.
 5. The digital picture display system ofclaim 1, further comprising a camera connected to the digital photoframe, wherein upon automatic detection with the camera of a recognizedviewer, the automated display module is adapted to automatically changephotos displayed on the digital display to photos from a correspondingelectronic device of the recognized viewer.
 6. The digital picturedisplay system of claim 1, further comprising a leveling deviceconnected to the frame, wherein the leveling device automaticallydetects when the frame is not level.
 7. The digital picture displaysystem of claim 5, further comprising an automated display adjustmentdevice in detection combination with the camera, wherein the automateddisplay adjustment device automatically detects frame environment and/orviewer position using the camera and automatically adjusts digitaldisplay settings.
 8. The digital picture display system of claim 1,further comprising a microphone in combination with the automateddisplay module and adapted to receive spoken instructions from a viewerto change the displayed photos.
 9. The digital picture display system ofclaim 5, wherein the automated display module includes or obtains aviewer profile of the recognized viewer and automatically changes thedisplay as a function of the viewer profile and the correspondingelectronic device upon detection of the recognized viewer through thecamera.
 10. The digital picture display system of claim 9, wherein theviewer profile is automatically learned and uploaded via a social mediaaccount of the recognized viewer.
 11. The digital picture display systemof claim 10, wherein the automated display module displays a slideshowof photos for the recognized viewer uploaded from at least one of: thecorresponding electronic device of the viewer or a social media accountof the recognized viewer or an electronic device of at least onecommunity member of the recognized viewer.
 12. The digital picturedisplay system of claim 11, wherein the automated display moduledisplays a slideshow of photos of the recognized viewer in a displayorder automatically determined as a function of time, location, and/orphoto content.
 13. The digital picture display system of claim 1,wherein the automated display module is adapted to automatically displayphotos associated by photo context with a recognized viewer andautomatically downloaded from social media accounts of a plurality ofcommunity members of the recognized viewer.
 14. The digital picturedisplay system of claim 13, further comprising a microphone incombination with the automated display module and adapted to receivespoken instructions from the recognized viewer to change the displayedphotos to a further social media account.
 15. The digital picturedisplay system of claim 1, further comprising a facial recognitionmodule in combination with the automated display module, wherein theautomated display module automatically changes the display upondetection of a recognized viewer with the facial recognition module. 16.The digital picture display system of claim 1, wherein the automateddisplay module augments a photo display on the digital display as afunction of photo metadata from the first or second electronic device ora social media account, wherein the photos displayed are automaticallydownloaded from a server to the digital frame.
 17. The digital picturedisplay system of claim 16, wherein the automated display modulesequences photos displayed on the digital display as a function ofprofiling traits selected from chronological order, photo location,photo activity, and/or community member.
 18. The digital picture displaysystem of claim 17, wherein each photo downloaded from the serverinclude a displayed tag identifying a photo location, a photo activity,and/or a present community member.
 19. The digital picture displaysystem of claim 1, wherein the automated display module is adapted toautomatically change photos displayed on the digital display to photosfrom the social media account of each of the first and second usersviewing the digital photo frame, upon automatic detection with thecamera of the first and second users viewing the digital photo frame.20. The digital picture display system of claim 1, wherein at least oneof: the first interaction application, the second interactionapplication, and the automated display module is configured toautomatically filter photos as a function of quality, wherein photos oflow quality are removed from the digital photo clusters or the digitalphoto frame.